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SECTION  1 
INTRODUCTION 


1.1  GENERAL 

This  is  the  third  formal  report  of  the  General  Research  Corpora¬ 
tion  (GRC)  on  the  ^Foreign  Military  Sales  (FMS)  Manpower  Projection 
Methodology"  project.  This  project  was  undertaken  for  the  Assistant 
Secretary  of  Defense  (Manpower,  Reserve  Affairs,  and  Logistics)  begin¬ 
ning  in  February  1979. 

1.2  OBJECTIVE 

The  objective  of  this  project  is  to  develop  a  standard  methodology 
for  forecasting  manpower  requirements  for  foreign  military  sales.  To 
accomplish  this  objective,  the  project  encompassed  three  tasks: 

1.  Improvement  of  the  current  Security  Assistance  Manpower 
Accounting  System  (SAMAS),  including  the  formulation  and 
recommendation  of  necessary  revisions  to  the  SAMAS  DoDI 
(1  Feb-31  May  1979). 

2.  Development  of  alternative  strategies  and  methods  for  fore¬ 
casting  future-year  FMS  manpower  requirements  (1  Feb-30  Sep 
1979). 

3.  Development  of  a  detailed  system  for  forecasting  future-year 
FMS  manpower  requirements,  including  the  testing  of  alterna¬ 
tives,  the  selection  and  development  of  a  preferred  method¬ 
ology,  and  the  preparation  of  implementing  instructions  for 
the  Services  (20  Feb-31  Dec  1980) . 

Although  this  project  was  intended  to  be  confined  to  FMS,  the  same 
forecasting  methodology  is  equally  adaptable  to  the  Military  Assistance 
Program  (MAP).  Accordingly,  the  alternative  methodologies  discussed  in 
this  report  encompass  both  FMS  and  MAP  manpower  requirements. 

1.3  PREVIOUS  REPORTS 

In  the  conduct  of  this  project,  we  have  submitted  two  Study  Docu¬ 
mentation  Reports  (SDR) : 
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•  Phase  I  SDR:  "Revised  SAMAS  Instructions"  which  covered 
our  review,  analysis,  and  findings  on  the  first  two  SAMAS 
reports,  our  recommended  revisions  to  the  draft  SAMAS  DoDI 
and  our  suggested  modifications  to  the  program  element 
structure  for  security  assistance  programs. 

•  Phase  II  SDR:  "Alternative  Methods  for  Forecasting  Security 
Assistance  Manpower"  which  covered  our  review,  analysis, 
experimentation,  and  findings  on  the  availability  of  useful 
data  and  alternative  methods  for  developing  program  projec¬ 
tions  and  manpower  forecasts. 

1.4  ORGANIZATION  OF  THIS  REPORT 

This  report  is  concerned  primarily  with  Task  3,  our  testing  and 
evaluation  of  alternative  methods  for  forecasting  security  assistance 
manpower  requirements.  In  Section  2,  we  describe  our  activities  and 
methodologies  for  test  and  validation.  In  Section  3,  we  describe  the 
results  of  our  tests  to  date.  In  Section  4,  we  present  our  conclusions 
on  completed  tests  and  our  recommendations  for  the  test  and  validation  of 
additional  methodologies  leading  to  the  selection  of  preferred  methodolo¬ 
gies  for  development  and  implementation. 

1.5  REPORTING  CONVENTIONS 

In  the  interest  of  clarity  and  simplicity,  we  have  adopted  three 
conventions  in  nomenclature  which  are  used  throughout  this  report: 

•  Although  the  Security  Assistance  Program  Includes  more 
activities  than  just  Military  Assistance  and  Foreign 
Military  Sales,  security  assistance  will  refer  to  the 
combination  or  aggregation  of  MAP  and  FMS  in  terms  of 
either  Sales  or  Deliveries. 

•  Although  the  Military  Assistance  Program  is  executed  in 
terms  of  MAP  Orders  and  Deliveries  and  the  Foreign  Military 
Sales  Program  is  executed  in  terms  of  FMS  Agreements  and 
Deliveries,  security  assistance  Sales  will  refer  to  both 
MAP  Orders  and  FMS  Agreements. 
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•  Although  the  terms  can  be  and  are  often  used  interchangeably, 
programs  will  always  be  projected  and  ^^gow^  will  always 
be  forecast. 


1.6  OUTSIDE  CONSULTANTS 

During  the  course  of  this  project,  we  have  engaged  the  services  of 
two  outside  consultants.  Both  are  acknowledged  experts  in  the  field  of 
statistical  methods  of  forecasting.  Dr.  Gulsseppi  A.  Forglonne,  Professor 
of  Management  Science  at  the  California  State  Polytechnic  University, 
-Inventoried,  rationalized,  and  selected  specific  statistical  methods  and 
software  packages  to  be  tested.  Dr.  Irwin  Greenberg,  Associate  Professor 
of  Business  Administration  at  George  Mason  University,  conducted  an  inde¬ 
pendent  critical  analysis  of  our  methodologies  and  the  results  attained. 
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SECTION  2 

TEST  AND  VALIDATION  ACTIVITIES 


2.1  GENERAL 

The  contractual  plan  for  Task  3,  Test  and  Analysis  of  Alternative 
Forecast  Methods,  prescribed  the  following  activities: 

•  Test  and  validate  program  projection  methodologies 

•  Test  and  validate  manpower  forecast  methodologies 

•  Select  and  recommend  preferred  methodologies 

2.2  BACKGROUND 

In  response  to  Congressional  and  GAO  criticisms  and  to  correct 
manpower  accounting  deficiencies.  General  Research  Corporation  (GRC) 
developed  and  the  Department  of  Defense  (DoD)  implemented  a  "Security 
Assistance  Manpower  Accounting  System  (SAMAS)."  It  provides  a  standard 
means  for  the  Military  Departments  and  Defense  Agencies  to  report  the 
numbers  of  personnel  engaged  in  security  assistance  activities,  the 
majority  being  in  the  Foreign  Military  Sales  (FMS)  program. 

In  an  early  review  of  SAMAS,^  the  GAO  cited  a  need  for  prescribing 
more  systematic  and  uniform  methods  for  the  Military  Services  to  program 
future  FMS  manpower  requirements.  An  effective  system  of  forecasting 
manpower  requirements  is  essential  for  DoD  to  obtain  the  necessary  man¬ 
power  ceiling  headroom  to  discharge  security  assistance  obligations. 

This  project  provides  for  the  development  and  ultimate  implementation  of 
such  a  manpower  forecasting  system.  This  report  presents  the  results  to 
date  of  our  test  and  validation  of  alternative  methodologies. 

2.3  RATIONALE 

In  our  earlier  research,  we  had  examined  the  Security  Assistance 
program  and  tested  the  historical  data  available  to  identify  various 
forecasting  methodologies  which  offered  reasonable  prospect  for  utility 


^GAO  Report  FGMDS  78-47,  Inadequate  Methods  Still  Used  to  Account  for  and 
Recover  Personnel  Costs  of  the  Foreign  Military  Sales  Program,  July  25, 


1978. 
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and  success.  Since  that  report  was  issued,  our  continuing  review  and 
experimentation  have  influenced  directly  the  sequence  and  procedures 
used  in  testing  and  validating  those  methodologies: 

•  Manpower  forecasting  is  at  least  a  two-step  procedure:  a 
program  or  workload  projection  and  an  associated  manpower 
requirement  forecast. 

•  Country  and  country  group  appear  to  have  little  value  as 
independent  variables  for  predicting  either  workload  or  manpower. 

•  Commodity  and  commodity  group  appear  to  have  great  value  as 
independent  variables  for  predicting  both  workload  and  manpower 
and,  indeed,  the  Service  and  activity  which  will  execute  the 
program. 

•  Sales  volume  (in  $  by  commodity)  appears  to  bear  most  directly 
on  administrative  manpower  requirements. 

•  Deliveries  volume  (in  $  by  commodity)  appears  to  bear  most 
directly  on  direct  manpower  requirements . 

•  The  MAP  component  of  Security  Assistance  is  small  in  comparison 
to  the  FMS  component.  The  projection  methodology  should  address 
the  whole  Security  Assistance  program  and  then,  the  components. 

Using  these  findings,  we  structured  our  test  and  validation  task 
plan  to: 

•  Construct  time  series  and  develop  forecast  equations  for 
sales  and  deliveries  for  FMS  and  MAP  (separately  and  combined) : 

(1)  Total  program 

(2)  Service  program 

(3)  Country  group  programs^ 

(4)  Commodity  group  programs 


*In  spite  of  the  apparent  lack  of  utility  as  a  forecast  variable,  country 
group  relationships  were  tested  for  three  reasons:  (1)  a  reluctance  to 
discard  any  methodology  without  adequate  test;  (2)  to  provide  program 
projections  in  conformance  with  the  strategic  orientations  of  the  Con¬ 
solidated  Guidance;  (3)  ready  availability  of  data. 
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•  Examine  results  of  test  and  validation  of  forecast  equations 
for  such  anomalies  as:  atypical  countries  or  country  groups; 
atypical  commodities  or  commodity  groups;  and  atypical 
Service  program  years. 

•  Identify  and  structure  the  essential  elements  of  program  pro¬ 
jection/program  guidance  by  both  level  and  direction  of  dis¬ 
aggregation: 


Program 

/  \ 

Service  Service 

Country  Group  Commodity  Group 

Country  Commodity 

•  Produce  all  essential  elements  of  program  guidance  using 
tested  and  validated  methodologies.  Obtain  for  each  method 
and  each  projection  a  coefficient  of  correlation  (or  deter¬ 
mination)  or  other  measure  of  accuracy  and  reliability. 

•  Construct  time  series  and  develop  macAO  forecast  equations 
for  security  assistance  manpower,  including:  sales  and 
deliveries  relationships  at  various  levels  of  program  pro¬ 
jection;  and,  manpower  estimating  equations  for  sales  and 
deliveries  at  various  levels  of  program  projection. 

•  Produce  macAO  forecast  of  manpower  requirements  using  tested 
and  validated  methodology  for  various  levels  of  program  sales 
and  deliveries  projection.  Develop  the  measure  of  accuracy 
and  reliability. 

•  Construct  a  compendium  of  Service  methods  for  estimating  man¬ 
power  requirements  based  on  workload  factors,  standards,  or 
measurements  to  be  adopted/adapted  for  use  as  iy\X,QAmzdJjOUt(l 
forecast  methods  for  various  levels  of  program  projection. 
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m  Analyze  each  method  for  accuracy  and  reliability  when  used 
by  the  developing  Service  for  its  primary  purpose,  and  for 
its  probable  utility  value  when  applied  to  other  uses/users. 

•  Identify  untested  or  undeveloped  methods  that  might  be  ap¬ 
plied  to  the  security  assistance  requirement. 

•  Produce  forecasts  of  manpower  requirements 

using  forecast  methodologies  identified  and  adopted.  Test 
and  validate  each  methodology  for  various  levels  of  program 
projection.  Develop  measures  of  accuracy  and  reliability 
for  each  methodology. 

•  Construct  historical  life  cycle  of  FMS  cases/MAP  orders  by 
Service  and  commodity  group.  Establish  key  events  in  the 
life  cycle  and  the  probabilities  and  rates  at  which  each 
event  occurs  and  is  completed. 

•  Establish,  from  DSAA  and  Service  records,  the  manpower  re¬ 
lationships  for  each  key  event  and  time  period  in  the  life 
cycle.  Develop  mi,QJiO  forecast  equations  for  security  assis¬ 
tance  manpower. 

•  Produce  VdicAO  forecast  of  manpower  requirements  using  tested 
and  validated  methodology  for  projecting  the  LOR  (case/order) 
life  cycle.  Develop  the  measure  of  accuracy  and  reliability. 

2.4  PROCEDURES 

The  DSAA  1979  Fiscal  Year  Series  Report  provides  30  years  (1950- 
1979)  of  individual  country  FMS  and  MAP  dollar  sales  and  deliveries  by 
year.  The  individual  country  data  were  posted  and  aggregated  into  country 
groups  corresponding  to  those  used  in  the  Consolidated  Guidance.  The 
country  group  sales  programs  were  then  summed  to  obtain  the  total  dollar 
value  of  security  assistance  (FMS  and  MAP  combined)  sales /orders  and  de¬ 
liveries  over  the  30-year  period.  Total  security  assistance  sales  and 
deliveries  data  were  used  to  derive  a  total  program  forecast  equation. 

The  aggregated  country  group  sales  and  deliveries  data  were  used  to 
derive  a  forecast  equation  for  sales  and  deliveries  by  country  group. 
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The  DSAA  FMS  and  MAP  data  files  provide  16  years  (1964-1979)  of 
detailed  data  to  form  a  time  series  of  dollar  sales  by  country  group  and 
Service.  These  data  were  aggregated  by  Service  and  used  to  develop  fore¬ 
casting  equations  for  dollar  sales  volume  by  Service  for  each  country  group. 

The  DSAA  FMS  and  MAP  Case  and  Detail  files  provide  16  years  (1964- 
1979)  of  detailed  data  to  form  a  time  series  of  dollar  sales  by  commodity 
group  and  Service.  These  data  were  aggregated  by  Service  and  commodity 
group  and  will  be  used  to  develop  forecasting  equations  for  dollar  sales 
volume  by  Service  for  each  commodity  group. 

The  LOR  subsystem  of  the  DSAA  Case  and  Detail  files  provides  only 
6  years  (1975-1980)  detailed  history  data  on  the  life  cycle  of  letters  of 
request.  These  data  are  being  aggregated  and  analyzed  to  develop  a  model 
of  the  LOR  process  by  Service  and  commodity  group. 

Using  aggregated  time  series  data  on  sales  and  deliveries,  a  rela¬ 
tionship  between  sales  and  deliveries  was  derived.  Based  on  the  assumption 
that  the  manpower  requirement  would  depend  on  both  total  dollar  sales  and 
total  dollar  deliveries,  a  manpower  relationship  was  also  developed  and 
tested.  These  relationships  were  used  to  derive  a  manpower  requirements 
forecast  equation  —  a  macro  method  for  manpower  forecast. 

Using  disaggregated  time  series  data  on  security  assistance  or  its 
components,  by  Service,  by  commodity  group,  and  by  country  group,  we 
have  begun  to  derive  workload  and  manpower  relationships  at  other  levels 
—  intermediate  methods  for  manpower  forecast. 

Using  disaggregated  time  series  data  from  the  LOR  subsystem  and 
Service  estimates  for  the  administrative  manpower  for  each  phase  in  the 
life  cycle,  we  will  derive  life  cycle  event  and  manpower  relationships  for 
administrative  manpower  —  a  micro  method  for  manpower  forecast. 
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2.5  DATA  ACCESS  AND  AVAILABILITY 

The  Defense  Security  Assistance  Agency  (DSAA)  maintains  a  rather 
comprehensive  file  of  historical  data  on  the  security  assistance  pro¬ 
grams  (countries,  commodities,  sales,  deliveries,  dollars).  The  Mili¬ 
tary  Services  and  Defense  Agencies  also  maintain  comprehensive  files  of 
historical  data  on  Service/Agency  conduct  of  security  assistance  programs 
(countries,  commodities,  sales,  deliveries,  dollars,  and  case  identity). 

There  is  no  historical  file  of  any  significance  (except  SAMAS) 
which  records  or  accounts  for  the  manpower  associated  with  the  security 
assistance  programs. 

2.5.1  Data  Repositories 

2. 5. 1.1  Defense  Security  Assistance  Agency  (DSAA) 

DSAA  maintains  its  computerized  records  for  FMS/MAP  transactions 
on  an  IBM  3033  computer  through  the  Director  of  OSD  Systems,  Air  Force 
Data  Systems  Center,  Defense  Security  System  Division  in  Room  4C653, 

The  Pentagon.  GRC  was  not  permitted  access  to  the  terminal  site  located 
in  Room  4C653.  In  lieu  of  working  with  DSAA  files  on  site,  DSAA  agreed 
to  furnish  both  hard  copy  and  tapes  to  GRC  for  use  as  discussed  below. 

1979  Fiscal  Year  Series  Report,  published  by  the  Data  Management 
Division,  Comptroller,  DSAA,  contains  data  in  dollars  for  fiscal  years 
1950-1979  by  country  group  for; 

•  FMS  agreements  (or  sales) 

•  FMS  deliveries 

•  MAP  program  (or  orders) 

•  MAP  deliveries 

These  data  were  useful  to  obtain  a  basic  or  macro  view  of  security  assis¬ 
tance  and  country  groups.  Data  on  country 'j^oup  totals  were  adjusted  to 
reflect  the  Consumer  Price  Index  (CPI)  and  entered  manually  into  the 
MULTICS  system  for  preliminary  analysis.  These  are  the  only  DSAA  data 
bhat  provide  a  full  30— year  record  at  the  requisite  level  of  disaggrega¬ 
tion,  Figure  2.1  is  a  graphic  presentation  of  the  30-year  Security  Assis¬ 
tance  Sales  and  Deliveries  program  in  constant  (FY  67)  dollars. 
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DSAA  furnished  GRC  two  MAP,  two  EMS,  and  one  LOA/LOR  data  tapes. 

•  Initially,  the  MAP  tapes  were  unreadable  on  MULTICS  because 
on  a  noncompatible  computational  field.  DSAA  reran  the 
tapes  with  the  required  conversions. 

•  No  problems  were  encountered  with  the  FMS  tapes;  however, 
a  problem  similar  to  the  MAP  problem  currently  exists 
within  the  LOA/LOR  tapes. 

•  Because  all  data  prior  to  1964  have  been  rolled  up  into  a 
1963  total,  only  16  years  (1964-1979)  of  program  data  are 
usable  for  analysis  and  forecasting. 

•  GRC  has  extracted  total  dollar  sales  data  for  the  FMS  and 
MAP  programs  by  year,  by  country,  and  by  Service. 

•  GRC  has  extracted  FMS  sales  and  MAP  orders  and  deliveries 
data  by  year,  by  commodity,  and  by  Service. 

2.5.1. 2  Office,  Assistant  Secretary  of  Defense  (Manpower,  Reserve 
Affairs  and  Logistics)  [OASD(MRA&L) ] 

OASD(MRA&L)  maintains  its  computerized  records  for  SAMAS  manpower 
data  through  the  Director  of  OSD  Systems,  Air  Force  Data  Systems  Center 
in  Room  2D279,  The  Pentagon.  Although  this  is  primarily  an  OASD(PA&E) 
MULTICS  facility,  MRA&L  and  GRC  have  access  to  terminals  for  the  develop¬ 
ment  and  maintenance  of  DoD  projects  and  files.  GRC  has  access  to  the 
MULTICS  on  the  MRA&L  activity  account. 

2. 5. 1.3  Georgetown  University 

Through  a  consultant  to  the  project  team,  GRC  has  obtained  access  to 
the  Georgetown  University  Data  Processing  Library.  Georgetown  University 
has  an  IBM  370  computer  with  DAS  360  terminals.  This  is  an  excellent 
facility  on  which  to  conduct  experimental  research,  create  a  data  base, 
and  test  methodologies.  A  software  system,  called  TROLL,  contains  sta¬ 
tistical  packages  well  suited  to  the  IMS  project.  GRC  has  used  the  facility 
extensively  since  mid-Jime. 
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2«5.2  Problems  and  Solutions 


Throughout  this  phase  of  the  project,  constrained  access  to  the 
data  and  the  data  processing  facilities  has  been  a  major  problem. 

•  In  early  April,  we  requested  transfer  of  the  DSAA  history 
files  to  the  MULTICS  site.  The  transfer  was  delayed  (until 
May)  awaiting  conversion  of  DSAA  data. 

•  In  the  interim,  we  were  able  to  experiment  with  annual  sales 
and  deliveries  data  by  manual  input  to  the  MULTICS  System. 
Initial  time  series  plots  were  made  and  the  Box- Jenkins 
statistical  package  was  rxin  using  these  data. 

•  In  late  May,  GRC  lost  its  computer  terminal  access  until  an 
MRA&L  funding  account  could  be  established  (in  June) . 

•  In  late  May,  DSAA  data  file  tapes  were  made  available  and 
retrieval  programs  were  written  to  read  the  tapes  and  extract 
the  useful  data. 

•  During  this  period,  the  team  was  limited  to  those  analyses 
which  could  be  accomplished  using  a  TI-59  programmable 
calculator.  Our  work  included:  a  number  of  time  series 
analyses  of  the  available  data;  several  least  squares  fits; 
and  exponential  smoothing  with  several  values  of  damping. 

•  By  mid-June,  the  team  had  begun  to  use  the  Georgetown  Univer¬ 
sity  computer  facility  where  our  project  consultant  had  ac¬ 
cess.  This  facility  has  terminals  linked  to  an  IBM  370 
computer  and  many  statistical  packages  designed  for  economic 
and  management  systems  analysis. 

•  In  late  June,  when  MULTICS  access  was  restored,  the  pre¬ 
viously  cited  computation  problems  with  DSAA  furnished  tapes 
were  identified. 

•  During  late  June  and  early  July,  the  team  again  used  manual 
data  input  to  form  files  of  sales  and  deliveries  data.  Ex¬ 
tensive  analysis  of  those  files  was  accomplished  using  the 
Georgetown  University  computer  facility. 
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•  By  late  July,  DSAA-fumished  FMS  tapes  were  finally  read 
and  extracts  produced  for  analysis  on  the  Georgetown  com¬ 
puter.  In  early  August,  MAP  card  decks  were  produced. 
Numerous  delays  were  experienced  during  this  data  transfer 
activity.  The  problems  appeared  to  be  anomalies  associated 
with  the  MCJLTICS  computer /card-punch  process. 

Even  without  these  problems  and  delays,  the  need  to  analyze  such  an 
immense  amoxint  of  data  has  presented  a  difficult  and  extremely  time- 
consuming  task.  Using  the  large  disk  files  and  easy-to-use  computer  pro¬ 
grams  available  at  Georgetown,  the  GRC  team  has  been  able  to  perform 
experiments  and  analyses  that  would  have  overloaded  the  available  memory 
space  on  the  MULTICS  system. 

2.6  METHODOLOGIES 

2.6.1  Time  Series 

Time  series  describe  the  occurrence  of  events  over  time.  The 
total  yearly  dollar  amounts  of  FMS  and  MAP  sales  and  deliveries  describe 
a  time  series.  Each  level  of  aggregation  of  FMS/MAP  sales  and  deliveries 
history  describes  a  separate  and  distinct  time  series. 

Time  series  models  use  past  history  to  predict  the  future  by  using 
the  derived  forecast  equation.  Time  series  data  are  required  for  a  suf¬ 
ficiently  long  period  of  time  to  ensure  confidence  that  any  observed 
trends  or  cycles  are  truly  part  of  the  data. 

When  a  single  time  series  treats  an  event  which  can  be  described  by 
summing  several  other  time  series,  all  time  series  must  be  consistent  and 
interrelated  to  provide  a  100%  certain  cause  and  effect  relationship. 

Thus,  the  sum  of  all  country  sales  and  deliveries  must  equal  the  total 
sales  and  deliveries  no  matter  how  many  subordinate  levels  of  aggregation 
(time  series)  are  examined. 
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From  our  extensive  tests  of  many  potential  forecasting  methods,  we 
have  identified  equations  which  permit  us  .to  establish  a  statistical  model 
to  forecast  the  total  dollar  sales  at  any  level  of  aggregation.  These 
statistical  models  do  not  pretend  to  forecast  the  individual  events  which 
will  cause  variations  in  sales.  Rather,  they  rely  on  the  fact  that  since 
an  event  with  a  particular  effect  has  occurred  in  the  past,  it  is  likely 
that  some  event  with  a  similar  effect  will  recur  in  the  future.  We  have 
developed  usable  techniques  to  make  such  forecasts. 

2.6.2  Disaggregations 

The  statements  about  time  series  made  above  hold  true  no  matter 
what  the  aggregation  level.  However,  we  have  found  many  differences  in 
time  series  characteristics  at  various  levels  of  disaggregation. 

It  appears  that  both  the  method  and  the  accuracy  of  the  forecast 
at  each  level  of  aggregation  will  differ.  This  may  militate  against 
attempting  to  forecast  sales  for  a  commodity  as  opposed  to  the  total 
commodity  group,  or  for  a  country  as  opposed  to  the  total  country  group. 

In  fact,  aggregation  is,  in  itself,  a  form  of  smoothing,  and  some  level 
of  aggregation  is  required  to  design  a  statistical  model.  Therefore,  we 
can  develop  forecasts  as  those  levels  of  aggregation  which  provide  con¬ 
sistent  and  satisfactory  results;  and  we  must  integrate  all  forecasts  to 
assure  their  overall  consistency. 

Although  time  series  presentations  are  used  in  all  cases,  the  formal 
forecasting  or  projection  equations  are  quite  different  and,  indeed,  more 
than  one  form  of  equation  may  be  used  to  forecast  the  same  event. 

2.6.3  Manpower  Relationships 

Like  the  sales-forecasting  relationship,  manpower  forecasting  is 
also  based  on  two  kinds  of  formulations:  cause  and  effect,  and  statistical 
relationships . 


There  are  two  forms  of  equations  used  to  describe  light  diffraction. 

One  is  based  on  a  theory,  the  other  on  statistics.  Both  are  correct 
and  are  used  at  different  times,  depending  on  the  detail  of  the  problem. 
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Cause  and  effect  models  are  usually  based  on  work  factors,  diffi¬ 
culty  indices,  and  methods  of  work.  They  range  from  rather  simple 
(single  variable)  productivity  models  established  for  repetitive  tasks 
to  more  complex  (multivariate)  models  which  consider  the  work  to  be  per¬ 
formed,  the  conditions  under  which  it  is  performed,  and  the  relationship 
of  the  work  to  other  work  performed  under  different  conditions. 

Statistical  models  are  usually  based  on  the  relationships  of 
easily  observable  quantities  to  manpower  requirements.  The  quantity  is 
not  usually  expressed  in  terms  of  work  units.  In  every  model  developed, 
however,  it  is  necessary  to  know  something  about  the  manpower  required 
and  to  relate  that  manpower  requirement  to  a  work  factor  or  a  statistical 
quantity.  In  the  present  instance: 

a.  There  are  limited  data  available  about  the  absolute  manpower 
required  .for  FMS  and  MAP  activities.  The  most  reliable  data  are  within 
the  SAMAS  system  but  have  been  accumulated  only  for  the  past  3  years. 

b.  There  are  even  less  data  available  about  the  relationship 
between  the  work  factors  used  in  productivity  models  and  the  work  factors 
involved  in  the  security  assistance  sales  process. 

c.  If  it  can  be  assumed^  that  security  assistance  deliveries  do 
not  differ  from  standard  deliveries,  there  are  considerable  data  avail¬ 
able  to  relate  security  assistance  delivery  activities  to  manpower  re¬ 
quirements.  Yet,  the  total  manpower  involved  in  the  delivery  activity 
represents  less  than  half  of  the  total  security  assistance  manpower 
requirement. 


There  is  some  argument  about  this  within  the  Army.  It  is  possible  that 
requirements  imposed  on  FMS-MAP  stocking  and  ordering  creates  a  different 
ratio  of  productive  work  for  FMS  and  MAP  than  for  normal  service  activity. 
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It  is  clear  that  both  cause  and  effect  models  and  statistical 
models  will  be  required  to  forecast  security  assistance  manpower  re¬ 
quirements  and  that  they  may  be  appropriate  to  different  levels  of  aggre¬ 
gation  .  Specif ically : 

a.  Administrative  workload  will  likely  be  forecast  using  both 
statistical  as  well  as  cause  and  effect  models. 

b.  Direct  workload  will  likely  be  forecast  using  cause  and 
effect  models.  These  models  will  likely  be  the  same  as  the  models  al¬ 
ready  used  to  estimate  other  Service  manpower  requirements  using  appro¬ 
priately  derived  modifications  to  reflect  the  differences  between  security 
assistance  and  "other"  (or  normal)  activities. 

It  appears  that  the  ongoing  test  and  validation  program  will  pro¬ 
vide: 

•  A  statistical  model  for  the  macro  method  of  manpower  fore¬ 
casting. 

•  A  combination  of  cause  and  effect  and  statistical  models  for 
intermediate  methods  of  manpower  forecasting. 

•  A  cause  and  effect  model  for  the  micro  method  of  manpower 
forecasting. 

2.7  MEASURES  OF  VALIDITY 

2.7.1  Data  Adequacy 

The  basic  mechanism  used  to  develop  macro  manpower  forecasts  is 
linear  regression.  Dollar  sales  and  dollar  deliveries  data  obtained 
from  the  DSAA  data  files  (or  derived  from  forecasts)  are  the  fundamental 
bases  of  the  regression  (i.e.,  the  independent  variables).  Manpower  data 
obtained  from  the  SAMAS  or  other  documentation  are  the  dependent  variables. 

Thus,  the  development  of  projections  for  dollar  sales  and  dollar 
deliveries  is  essential  to  produce  a  forecast  of  manpower  requirement, 
and  the  following  discussion  focuses  on  the  adequacy  of  the  data  to  pro¬ 
vide  those  projections. 
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Thirty  years  of  "adjusted"  data  for  dollar  sales  and  dollar  deliveries 
by  country  are  available  within  the  DSAA  Fiscal  Year  Series  Report.  No 
other  data  are  known  at  comparable  levels  of  reporting.  The  time  period 
for  which  data  are  available  spans  the  total  life  of  the  FMS  and  MAP 
programs.  These  data  are  considered  to  be  both  complete  and  accurate. 

Sixteen  years  of  data  are  available  within  DSAA  history  files  which 
detail  the  transactions  which  aggregate  to  total  dollar  sales  and  deliveries. 
These  files  include  information  on  cases,  case  loads,  and  the  specific 
numbers  of  commodity  items  ordered  and  delivered  by  Service.  These  data 
are  also  considered  accurate  since  the  file  is  a  subset  of  the  total  30- 
year  data  base. 

Manpower  data  are  contained  only  within  the  SAMAS  data  base.  Man¬ 
power  data  have  been  reported  in  detail  for  only  3  years.  It  is  difficult 
to  judge  the  accuracy  of  the  data  but  SAMAS  is  all  that  is  available  and 
represents  the  official  position  of  the  Services  with  respect  to  security 
assistance  manpower. 

The  sales  and  deliveries  data  used  in  this  program  to  represent 
the  independent  variables  are  generally  accurate  and  contain  sufficient 
detail  to  consider  multiple  levels  of  aggregation. 

•  30  years  of  program  data  at  the  highest  levels  of 
aggregation  are  adequate. 

•  16  years  of  program  data  at  lesser  levels  of  aggregation 
are  only  marginal. 

The  manpower  data  available  to  this  program  to  represent  the  de¬ 
pendent  variables  must  be  accepted  as  reasonably  accurate. 

•  3  years  of  manpower  data  at  any  level  of  aggregation  are 
grossly  inadequate. 

2.7.2  Data  Applicability 

Work  to  date  has  considered  the  dollar  volume  of  sales  and  de¬ 
liveries  to  be  the  basic  measure  of  FMS-MAP  activity.  Typically, 
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measures  of  work  are  stated  in  terms  other  than  dollars.  Thus,  the 
relationships  we  are  developing  are  at  a  different  level  than  the  cause 
and  effect  models  which  use  work  factors. 

There  are  methods  available  to  develop  a  relationship  between 

dollar  volume  and  work  factor.  The  usual  method  is  to  divide  the  dollar 
value  by  the  number  of  units  involved  and  use  the  imputed  value.  When 
that  is  done,  a  forecast  of  dollars  can  be  made  and  converted  to  units 
of  work.  In  this  case,  there  are  many  commodities  involved.  Within 
each  commodity,  there  are  many  items.  The  possibility  of  computing 
directly  an  average  value  of  dollars  per  a  commodity  into  a  work  unit 
or  work  factor  depends  upon  an  assumption  of  a  constant  item-quantity 
mix  over  the  time  period  of  concern.  This  would  be  unlikely. 

2.7.3  Exogenous  Factors 

The  forecasting  equations  for  manpower  are  based  on  the  formula¬ 
tion  of  sales  program  projection  equations  using  time  series  sales  data. 
Exogenous  events  are  not  specifically  identified  nor  are  the  dimensions 
of  their  effect  considered.  The  basic  philosophy  is  that  such  types 
of  events  will  continue  to  occur  at  similar  rates  and  with  similar 
effects  into  the  future.  The  validity  of  this  assumption  can  be  assessed 
only  over  time.  At  the  moment,  the  data  span  may  be  inadequate  to  check 
the  assumption. 

2.7.4  Accuracy 

The  regression  statistics  which  measure  the  accuracy  of  a  sales 
projection  are  the  first  indication  of  error.  The  regression  statis¬ 
tics  which  measure  the  accuracy  of  a  deliveries  projection  are  a 
second  indication  of  error.  The  manpower  forecast  will  be  made  using 
either  the  sales  projection  or  a  sales  and  deliveries  projection.  The 
regression  statistics  which  measure  the  accuracy  of  the  manpower  fore¬ 
cast  are  the  ultimate  indication  of  error. 
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2*7.5  Confidence 


Since  there  are  a  minimum  of  three  steps  in  producing  a  macro 
manpower  forecast,  the  manpower  estimates  are  affected  by  at  least  three 
errors : 

•  The  error  in  making  the  sales  projection. 

•  The  error  in  making  the  deliveries  projection. 

•  The  error  in  making  the  manpower  forecast  from  the  sales 

and  deliveries  projections. 

The  total  error  in  a  manpower  projection  then  is  the  cxmulative  effect  of 
the  errors  made  in  each  step  along  the  way. 

2.7.6  Comparison 

In  deciding  among  forecasting  alternatives,  there  is  a  clear  re¬ 
quirement  to  minimize  the  difference  between  the  data  that  are  forecast 
and  the  actual  data  that  are  observed  later. 

•  When  a  single  regression  relationship  is  used,  the  measure 

2 

of  utility  is  a  coefficient  of  determination  (r  ) ,  which 
indicates  the  percentage  of  variation  in  the  dependent 
variable  explained  by  the  regression  equation. 

•  When  more  than  one  regression  relationship  is  used,  the 
requirement  to  minimize  the  errors  applies  to  the  entire 
process.  This  might  result  in  the  selection  and  use  of 
alternative  regression  equations. 


2.7.7  Results 

The  results  of  our  test  and  validation  activities  are  described  in 
Section  3  following. 
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SECTION  3 

TEST  AND  VALIDATION  RESULTS 


3.1  GENERAL 

The  contractual  plan  for  Task  3,  Test  and  Analysis  of  Alternative 
Forecast  Methods,  requires  that  the  results  of  our  tests  and  analyses 
be  documented  to  include  such  details  as:  the  nature,  source,  and 
availability  of  data  used;  the  sample  size  or  similar  statistical  infor¬ 
mation  for  each  alternative  tested;  and  the  coefficient  of  determination 
or  other  measures  of  accuracy/ re liability  obtained. 

3.2  PROGRAM  PROJECTIONS 

In  our  Phase  II  report,  we  stressed  the  need  for  a  suitable  program 
projection  as  a  basis  for  a  manpower  forecast.  Most  logically,  that  pro¬ 
jection  would  be  developed  and  published  by  DSAA  as  future  year  program 
guidance.  The  guidance  should  be  reasonably  accurate,  should  be  coor¬ 
dinated  with  the  Department  of  State,  and  should  provide  a  sufficiently 
definitive  basis  on  which  the  Service  and  Agencies  can  forecast  and  pro¬ 
gram  their  future  year  manpower  requirements.  The  ultimate  importance 
of  a  valid  and  useful  program  projection  was  confirmed  in  the  Service 
comments  to  our  Phase  II  report  and  then  affirmed  by  OASD(MRA&L)  on 
6  June  1980.  Accordingly,  the  major  part  of  our  test  and  validation  ac¬ 
tivities  to  date  have  concentrated  on  the  establishment  of  methodologies 
for  making  program  projections  with  a  high  order  of  accuracy,  validity, 
and  utility. 

3.2.1  Program  Guidance 

Rather  than  let  the  available  data  lead  us  into  a  form  of  possible 
program  projection,  we  examined,  in  some  detail: 

•  The  kinds  of  program  guidance  now  provided  to  the  Services 
(where,  when,  and  in  what  detail). 

•  The  kinds  of  program  guidance  now  desired  by  the  Services 
(where,  when,  and  in  what  detail). 
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From  these  detailed  reviews,  both  of  guidance  docximents  and  of 
Service  requirements,  we  have  concluded  that  the  basic  program 
guidance  for  security  assistance  programs  should  be  provided  in  the 
Consolidated  Guidance.  It  is  published  annually  to  set  the  strategic 
objectives  of  DoD  over  the  five-year  period  beyond  the  Budget  Year.  The 
program  guidance  should  provide  the  level  of  security  assistance  sales 
activity  by  Service  or  Agency  in  terms  of  the  five  strategic  geographic 
regions  of  interest  in  the  Consolidated  Guidance.  Figure  3.1  is  an 
example  of  form  and  content  for  the  security  assistance  program  guidance 
to  be  provided. 

This  level  and  form  of  program  guidance  will  satisfy  the  require¬ 
ment  for  the  security  assistance  program  to  be  responsive  to  and  in 
support  of  the  overall  DoD  strategy  for  the  period.  It  could  also, 
satisfy  the  Service  requirement  to  use  a  vnevo  method  of  manpower  fore¬ 
cast.  It  will  not  satisfy  the  Service  requirement  for  sufficient  guidance 
on  which  to  develop  a  manpower  program  in  support  of  security  assistance 
activities.  At  least  one  additional  level  of  program  guidance  is  required. 

The  additional  guidance  could  also  be  published  in  the  Consolidated 
Guidance.  More  likely,  it  would  be  developed  for  and  published  in  the 
annual  Administrative  Budget  Call  issued  by  DSAA  directly  to  the  Services. 

This  guidance  should  probably  address  each  Service's  security  assistance 
program  in  terms  of  a  sales  program  (in  $);  a  deliveries  program  (in  $); 
and  a  case  program  (in  #  of  cases)  for  each  commodity  group  within  the 
Service  program.  Figure  3.2  is  an  example  of  form  and  content  of  the 
security  assistance  program  guidance  to  be  provided  to  each  Service  and  Agency. 

This  level  and  form  of  program  guidance  would  satisfy  the  basic 
requirement  for  detail  necessary  for  the  Service  to  develop  its  own 
estimate  of  specific  workload  and  the  related  manpower  requirements  using 
either  'vn'bBmBd'Lcii^e  or  nwcxco  methods  for  the  manpower  forecast. 
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FY  1983 


SECURITY  ASSISTANCE  SALES  PROGRAM 
(in  $  millions) 


DEFENSE 

ARMY 

NAVY 

AIR  FORCE 

OTHER 

TOTAL  PROGRAM 

xxxx 

XXX 

XXX 

XXX 

XXX 

WESTERN  EUROPE  AND  NATO 

XXX 

XXX 

XXX 

XXX 

XXX 

NEAR  EAST  AND  SOUTH  ASIA 

'  XXX 

XXX 

XXX 

XXX 

XXX 

EAST  ASIA  AND  PACIFIC 

XXX 

XXX 

XXX 

XXX 

XXX 

LATIN  AMERICA 

XXX 

XXX 

XXX 

XXX 

XXX 

AFRICA 

XXX 

XXX 

XXX 

XXX 

XXX 

Figure  3.1.  Consolidated  Guidance 


FY  1983 

SECURITY  ASSISTANCE  PROGRAM 

-  AEMY 

COMMODITY  GROUP 

SALES  ($) 

DELIVERIES  ($) 

CASES  (#) 

AIRCRAFT 

* 

XXX 

XXX 

XXX 

COMBAT  VEHICLES 

*  *  * 

XXX 

XXX 

XXX 

TRAINING 

XXX 

XXX 

XXX 

TOTAL 

Figure  3.2.  Service  Guidance 
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3.2.2  Program  Data  Characteristics 

We  have  found  many  differences  in  time  series  characteristics  at 
various  levels  of  disaggregation: 

•  The  total  security  assistance  (FMS.  +  MAP  combined)  time 
series  displays  two  major  cycles.  The  first  cycle  is 
of  ^22  years  duration  and  is  complete.  The  second  is  of 
only  ^8  years  duration  and  is  incomplete. 

•  Each  country  group  time  series  displays  discrete  but 
different  sales  cycles.  In  fact,  the  country  groups 
(except  "Western  Europe  and  NATO")  indicate  only  one 
cycle  and  these  at  dissimilar  times  between  country 
groups.  The  "Western  Europe  and  NATO"  country  group 
displays  two  cycles  similar  to  the  total  program. 

•  Individual  country  time  series  often  show  zero  sales 
volumes  at  various  times.  Typically,  fewer  than  20 
countries  (of  about  75  countries  in  each  annual  program) 
account  for  at  least  90%  of  the  total  dollar  (fms  +  MAP 
combined)  sales  in  each  annual  program. 

•  Service  time  series  display  multiple  cycles  which  differ 
in  time  phasing  among  the  Services. 

•  Commodity  group  time  series  display  cycles  different  than 
those  seen  in  either  the  country  group  or  the  Service  time 
series. 

The  best  forecast  methods  for  one  level  of  aggregation  did  not  always 
suit  another.  Thus,  because  sales  forecast  equations  developed  and  tested 
for  varioxas  levels  of  aggregation  took  two  different  forms,  forecast  equa¬ 
tions  were  defined  for  each  change  of  component  or  level  of  aggregation 
(i.e..  Service,  countries,  commodities  within  either  a  country  group  or  a 
commodity  group) . 

3.2.3  Program  Projection  (Sales) 

Program  projections  have  been  made  for  the  total  volume  of  sales 
at  several  levels  of  aggregation: 
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•  Total  Security  Assistance  programs 

•  FMS  program 

•  MAP  program 

•  Country  Group  program 

•  Service  program 

•  Commodity  Group  program 

Sales  projections  are  based  on  the  regression  of  dollar  sales  volume 

over  time  and  the  regression  equations  take  two  forms: 

•  A  logarithmic-exponential  form 

•  A  ciomulative  sales  form 

Total  sales  projections  are  based  on  the  existence  of  two  cycles 
within  the  period  1950-1979,  each  cycle  of  ^22  years  duration.  On  this 
basis,  the  data  from  1973  onward  were  forecast  based  on  the  form  of  the 
equations  derived  from  the  data  base  1950-1972. 

The  Hoerl’s  Special  Function,  a  logarithmic-exponential  form  of 
projection  equation,  considers  the  data  in  three  periods: 

•  A  buildup  and  partial  decay,  1951-1960 

•  A  period  of  decay,  1961-1970 

•  A  buildup  and  partial  decay,  1971-1979 

The  Hoerl^s  Special  Function  equation  takes  the  form: 
log  Sales  =  log  a^  +  b^(log  Time)  +  c^(Time) 

Figure  3.3  is  a  plot  of  actual  FMS  and  MAP  sales  experience  in  con¬ 
stant  dollars  for  the  period  1951-1979  against  a  forecast  of  sales  over 
that  period  using  Hoerl’s  Special  Function  with  constants  derived  from 
regression  analysis  in  three  phases.  The  vertically  shaded  area  is  the 
difference  between  the  actual  and  the  Hoerl^s  forecast  sales  volumes. 
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Figure  3.3  Total  Annual  Sales  (Actual)  vs  Total  Annual  Sales 
(Forecast),  1950-1979.  (Using  the  Hoerl’s  Special  Function) 


Hoerl’s  regression  statistics  for  the  three  periods  were: 

•  1950-‘1959  -  coefficient  of  determination  =  ,75;  standard 
error  in  Sales  =  691  million  ('^J23%  of  the  average  yearly 
sales  program) 

•  1960-1969  -  coefficient  of  determination  =  .863;  standard 
error  in  Sales  =  221  million  (^9.9%  of  the  average  yearly 
sales  program) 

•  1970-1979  -  coefficient  of  determination  *  .8995;  standard 
error  in  Sales  =  1,592  million  {'^>29%  of  the  average  yearly 
sales  program) 

The  cumulative  equation  form  of  program  projection  considers  the 
data  in  only  two  periods: 

•  A  complete  cycle,  1951-1973 

•  An  incomplete  cycle,  1974-1979 

The  cumulative  equation  takes  the  form: 

Cumulative  Sales  =  +  b^(Time) 

Figure  3.4  is  a  plot  of  actual  FMS  and  MAP  sales  experience  in 
constant  dollars  for  the  period  1951-1979  against  a  forecast  of  sales 
over  that  period  using  the  cumulative  sales  model  regressed  in  two  periods. 
The  horizontally  shaded  area  is  the  difference  between  the  actual  expe¬ 
rience  and  the  cumulative  sales  model  forecasts.  For  the  cumulative  sales 
equation,  the  regression  statistics  for  the  two  periods  were: 

•  1950-1972  -  coefficient  of  determination  =  .98819; 
standard  error  in  Sales  =  1,810  million  (^^^3.2%  of  the 
cumulative  sales  over  the  period) 

m  1973-1979  -  coefficient  of  determination  =  .98739; 

standard  error  in  Sales  =  1,960  million  ("^4.1%  of  the 
cumulative  sales  over  the  period) 
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CONSTANT  DOLLARS  (IN  BILLIONS) 


Figure  3.4  Total  Sales  (Actual)  vs  Total  Sales  (Forecast) 
1950-1979.  Cumulative  Method. 


A  comparison  of  the  plots  provided  by  Hoerl's  and  the  ctomulatlve 
sales  model  reveals  the  following: 

•  The  cumulative  sales  model  follows  the  form  of  the  cumulative 
sales  experience  well,  but  the  individual  yearly  sales  values 
obtained  using  the  model  have  considerable  difference  from 
actual  values. 

•  The  Hoerl's  approximation,  because  it  is  a  smoothed  model, 
provides  less  difference  between  projected  and  observed  yearly 
values  of  sales,  but  does  not  reproduce  the  sales  curve  in 

as  much  detail  as  does  the  cumulative  sum  model. 

For  the  purpose  of  projecting  absolute  dollar  yearly  sales,  the 

Hoerl' s  approximation  appears  better  than  the  cumulative  sum  model.  For 

obtaining  trends  and  cumulative  values  against  which  to  adjust  the  smoothed 

1 

Hoerl's  estimates,  the  cumulative  model  is  best. 

Details  of  these  regression  analyses  for  the  total  program  portion 
of  the  program  projection  activity  are  in  Appendix  A. 

Figure  3.5  is  a  plot  of  the  cvnnulative  FMS  and  MAP  sales  experience 
in  constant  dollars  for  the  16-year  period  1964-1979  for  the  Western 
Europe  and  NATO  country  group.  Constants  were  derived  from  two  separate 
regressions:  the  first  for  1964—1972;  the  second  for  1973—1979.  Again, 
the  shaded  area  represents  the  difference  between  actual  sales  history 
and  the  forecast  of  sales.  For  the  regression  period  1964-1972,  the 
coefficient  of  determination  was  .98277  and  the  standard  error  was  300 
million  of  the  cumulative  sales  over  the  period).  For  the  regres¬ 

sion  period  1973-1979,  the  coefficient  of  determination  was  .94036  and 
the  standard  error  was  966  million  ('v>9.5%  of  the  cumulative  sales  over 
the  period) . 


^The  cumulative  model  is  analogous  to  a  "trend  line"  in  "classical  de¬ 
composition";  the  Hoerl's  model  is  analogous  to  a  "cyclic"  or  "repetitive 
variation"  from  the  trend  line. 
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CONSTANT  DOLLARS  (IN  BILLIONS) 


•  ACTUAL  SALES  1964-1972 
◄  FORECAST  SALES  1964-1972 
■  ACTUAL  SALES  1973-1979 

♦  FORECAST  SALES  1973-1979 


Figure  3.5  Cumulative  Actual  Sales  vs  Cumulative  Forecast  Sales 
Western  Europe  and  NATO  Goxintry  Group 
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Details  of  these  regression  analyses  as  well  as  regression  statistics 
and  plots  for  all  country  groups  are  in  Appendix  B. 

Figure  3.6  is  a  plot  of  cumulative  Army  FMS  and  MAP  total  sales 
experience  in  constant  dollars  for  the  period  1964-1979  againstrthe  forecast 
of  FMS  and  MAP  sales.  Constants  were  derived  from  two  separate  regressions. 
Once  again,  the  shaded  area  represents  the  differences  between  the  forecast 
and  actual  experience.  For  the  regression  period  1964-1972,  the  coeffi¬ 
cient  of  determination  \<ras  .9929  and  the  standard  error  was  133  million 
('^^2.6%  of  the  cumulative  sales  over  the  period).  For  the  regression 
period  1973-1979,  the  coefficient  of  determination  was  .9747  and  the 
standard  error  was  1,140  million  ('\^6.2%  of  the  cumulative  sales  over  the 
period) . 

Details  of  these  regressions  for  the  Army  and  the  regression  statis¬ 
tics  and  plots  for  all  other  Services  are  in  Appendix  C. 

Details  of  the  regressions  for  Commodity  Group  and  the  regression 
statistics  and  plots  are  in  Appendix  C. 

3.2.4  Program  Projection  (Deliveries) 

Program  projections  have  also  been  made  for  the  total 
security  assistance  deliveries.  Total  deliveries  projections  are  based 
either  on  the  existence  of  two  distinct  deliveries  cycles  within  the  30- 
year  period  or  on  a  derived  relationship  between  sales  and  deliveries. 

Deliveries  projections,  based  on  the  regression  of  dollar  deliveries 
volume  over  time,  take  a  logarithmic-exponential  form: 

log  Deliveries  =  log  a^  +  b^(log  Time)  +  c^(Time) 

Deliveries  projections,  based  on  the  sales  projections,  use  a  model 
in  the  form: 


3-11 


•  ACTUAL  SALES  1964  - 1972 
◄  FORECAST  SALES  1964  - 1972 
■  ACTU  A  L  SA  LES  1 973  -  1 979 

♦  FORECAST  SALES  1973  -  1979 


1980 


Sales  vs  Cvimulative  Forecast  Sales  -  Army 


L2 


(year  i-1) 


Deliveries i)  ’  »1  +  ‘itS^es)  +  c^(Sales) 

+  dj (Sales)  ^.2)  +  ej^(Sales)  ^.3,  +  f^CSales) 

Figure  3.7  is  a  plot  of  total  FMS  and  MA.P  deliveries  in  constant 
dollars  against  a  forecast  of  deliveries  using  this  model.  The  1950-1979 
time  period  was  divided  into  two  cycles,  and  regression  analyses  were 
performed  for  each  cycle.  Once  more,  the  shaded  areas  show  the  difference 
between  forecast  deliveries  and  actual  deliveries  experience.  For  the 
first  period,  the  coefficient  of  determination  was  .785  and  the  standard 
error  was  520  thousand  (^237o  of  the  average  yearly  deliveries.  For  the 
second  period,  the  coefficient  of  determination  was  .9577,  and  the  stan¬ 
dard  error  was  318  thousand  (^11%  of  average  yearly  deliveries). 

Details  of  the  regression  and  plots  of  the  deliveries  are  in 
Appendix  E. 

To  date,  deliveries  can  be  forecast  with  confidence  for  the  overall 
deliveries  program.  The  application  of  these  equations  to  disaggregated 
data  sets  has  not  yet  been  validated. 

Figure  3.8  is  a  plot  of  actual  Sales  and  Deliveries  over  the  30-year 
period  and  a  projection  of  Sales  and  Deliveries  to  1983  using  the  Hoerl’s 
Special  Function  method  for  sales^  and  the  cause  and  effect  model  for 
deliveries. 

3.3  A  MACRO  METHOD  OF  MANPOWER  FORECAST 


3.3.1  Program  Projection 

Conceptually,  a  gross  or  macro  method  of  manpower  forecast  might  be 
based  on  the  establishment  of  a  functional  relationship  of:  total  sales 


^The  relationship  between  the  cyclical  component  of  sales  (obtained 
tising  Hoerl's)  was  adjusted  to  conform  to  the  trend  line  (derived  using 
the  cumulative  method) . 
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CONSTANT  DOLLARS  (IN  BILLIONS) 
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to  total  manpower;  or,  total  deliveries  to  total  manpower;  or,  total 
cases  to  total  manpower.  On  the  assimption  that  sales  or  number  of  cases 
probably  drive  the  administrative  manpower  (about  half)  and  deliveries 
probably  drive  the  remainder  (about  half)  we  performed  a  number  of  regres¬ 
sion  experiments.  Ultimately,  we  established  a  reasonable  and  useful 
relationship  between  total  sales  and  total  deliveries.  The  generalized 
equation  takes  the  form  (1) : 

“83  =  ^<S83>  *  ^  +  «S80>  + 

This  relationship  varies  somewhat  from  Service  to  Service,  from 
comtry  to  country,  and  from  commodity  to  commodity.  But,  for  a  macro 
method  (Total  program  to  Total  manpower)  this  relationship  is  valid. 

We  then  performed  a  number  of  other  experiments  to  determine  the 
relationship  between  total  manpower  (M)  and  the  levels  of  total  sales  (S) 
and  total  deliveries  (D) .  We  again  established  a  reasonable  and  useful 
relationship  which  takes  the  form  of  generalized  equation  (2) : 

(2)  Mj3  -  a  +  8(833)  +  c(D33) 

Finally,  since  deliveries  (D)  in  any  year  are  a  function  of  sales 
(S)  in  that  year  and  sales  in  a  number  of  preceding  years,  we  substituted 
the  relationship  between  sales  and  deliveries  developed  in  equation  (1) 
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3.3.2  Manpower  Forecasts 

Attempts  to  produce  a  macro  method  for  manpower  forecasting  are 
severely  limited  by  the  data  available  in  the  SAMAS  reports.  The  total 
SAMAS  data  are  simnnarized  in  Figure  3.9. 


Date  of 
Report 

3/30/78 

4/18/78 

1/25/79 

5/31/79 

11/16/79 

1/14/80 


1977  MY 
21843 
21758 


1978  MY 

24565 

24566 


1979  MY 
25406 
25432 
29621 
28453 
25741 
23049 


1980  MY 
25694 
25787 
29709 
26900 
24552 
23507 


1981  MY 


29469 

26670 

24208 

22741 


1982  MY 


22909 

22283 


Figure  3.9.  Security  Assistance  Manpower  (in  man-years) 


Only  3  years  of  actual  and  3  years  of  estimated  data  are  available. 
Three  sets  of  manpower  data  were  formed: 

9  A  set  of  forecast  man-years  data  which  represent  the 
Service  estimates  of  manpower  requirements. 

9  A  set  of  data  of  3  years'  actual  reported  manpower  and 
3  years  of  "average"  estimated  manpower  requirements.^ 

9  A  set  of  3  years  of  actual  data  and  3  years  of  "latest" 
estimate. 


The  estimated  man-years  for  each  year  were  averaged:  for  1981,  four 
estimated  man-year  values  were  added  together  and  the  sum  divided  by 
four. 
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The  three  sets  of  data  were  used  to  derive  sets  of  regression  con¬ 
straints  in  the  equation; 

Manpower  =  a^  +  (Sales)  +  (Deliveries) 

In  the  actual  regression  test,  the  coefficient  of  determination 
was  1.0  and  the  standard  error  was  .0135  man  years.  These  values  describe 
a  perfect  fit  between  the  data  and  the  tested  equation.  Appendix  F  de¬ 
tails  the  manpower,  sales,  and  deliveries  data  used  in  this  regression. 

3.3.3  Forecast  Accuracy 

From  these  regressions,  it  can  be  seen  that  although  the  coefficient 
of  determination  of  the  regression  is  remarkably  high  (1.0),  there  is  a 
wide  variation  in  the  regression  constants  obtained  for  each  data  set. 
Clearly,  these  attempts  indicate  that  even  though  the  form  of  the  re¬ 
gression  equation  is  likely  to  be  correct.,  there  is  insufficient  data 
to  make  a  believable  forecast. 

There  appears  to  be  some  utility  in  the  macAO  forecasting  technique 
if  and  when  enough  years  of  manpower  data  are  available  to  develop  a 
predirective  equation.  For  the  present,  however,  we  believe  it  to  have 
only  limited  value.  Perhaps  its  main  use  will  be  to  provide  bounds  to 
check  manpower  totals  against  the  incremental  manpower  subtotals  forecast 
by  other  methodologies. 

3.4  INTERMEDIATE  METHODS  OF  MANPOWER  FORECASTS 


3.4.1  Program  Projection  (Sales  and  Deliveries  by  Commodity) 

IntoArntdicutz  manpower  forecasting  methodologies  can  be  based  on 
the  projection  of  service/ commodity  sales  and  deliveries  programs  in 
combination  with  a  work  unit  conversion. 
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Using  SAMAS  data: 

•  We  can  apportion  the  total  manpower  of  a  Service  among  its 
organizational  elements . 

•  We  can  associate  the  activity  of  each  organizational  element 
with  a  particular  commodity  set. 

•  We  can  project  the  sales  and  deliveries  of  that  commodity 
set  to  develop  the  manpower  forecasting  methodology. 

3.4.2  Manpower  Forecasts 

At  the  subordinate  command  levels,  manpower  requirements  appear  to 
be  quite  dependent  on  workloads.  While  a  statistical  model  of  the  manpower 
associated  with  commodity  sales  may  indicate  a  gross  or  average  level  of 
manpower  requirement,  the  determination  of  the  direct  case  and  administra¬ 
tive  manpower  requirements  would  be  better  made  using  a  cause  and  effect 
(or  workload)  model. 

As  a  first  test,  we  will  regress  the  Army  sales  and  deliveries  against 
Army  manpower.  Then,  we  will  regress  Army  commodity  sales  and  deliveries 
against  Army  manpower  levels  within  the  commands  with  major  responsibilities 
for  sales  and  deliveries  of  the  commodities. 

3.5  A  MICRO  METHOD  OF  MANPOWER  FORECAST 

3.5.1  Program  Projection  (Workload) 

Miovo  manpower  forecasting  requires  a  cause  and  effect  relationship 
between  workload  and  manpower.  It  assumes  that  a  manpower  loading  equation 
can  be  developed  to  permit  an  estimate  of  manpower  based  on  an  estimate  of 
workload. 

•  The  Air  Force  uses  estimating  equations  routinely.  For 
example,  the  Air  Force  Logistics  Command  (AFLC)  MANSAP 
model  provides  estimating  equations  for  manpower  require¬ 
ments  based  on  total  requisitions  anticipated  as  modified 
by  probable  country  sales. 
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•  The  Army  Logistic  Center  has  developed  a  simulation  model 
(LOGCTR)  which  forecasts  administrative  manpower  based  on  a 
series  of  estimated  workloads  which  are  generated  sequen¬ 
tially.  Workloads  consist  of  LORs,  LOOs,  LOAs,  requisitions, 
billings,  number  of  cases,  and  number  of  lines. 

Comparisons  between  the  Air  Force  and  Army  models  reveal  consider¬ 
able  difference  between  the  workload  details  necessary  to  exercise  the 
models.  We  will  review  the  existing  DSAA  LOR  data  files  to  determine 
how  much  detail  is  available  to  support  workload  forecasts  at  various 
levels.  The  steps  in  our  analysis  will  be  as  follows: 

•  The  total  LOR  file  will  be  used  to  derive  total  file 
statistics  by  year;  the  ntmiber  of  LORs  initiated  each 
month  during  the  years;  the  nvimber  of  LOOs  outstanding 
each  month  during  the  year;  the  number  of  LOAs  outstanding 
each  month  during  the  year. 

•  LORs,  LOOs,  and  LOAs  will  be  categorized  by  Service, 
commodity,  and  country.  The  total  number  of  dollars 
represented  by  all  LOR,  LOO,  LOA,  and  the  number  of  items 
represented  by  those  dollars  will  be  associated  with  each 
LOR,  LOO,  and  LOA. 

•  Individual  LORs  will  be  tracked  to  determine  the  probability 
that : 

an  LOR  becomes  an  LOO; 
an  LOO  becomes  an  LOA; 
an  LOR  becomes  an  LOA. 

The  association  of  Service/commodity/number  of  items,  and  the  total 
dollar  value  will  permit  development  of  a  four  dimensional  probability 
distribution  function  and  the  computation  of  dollar  value  commodity  mix 
ranges.  All  of  these  probabilities  will  have  a  time  dimension.  That  is, 
the  length  of  time  it  takes  for  each  transition  step  to  occur  will  be  used 
to  stratify  the  transition  probabilities. 
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m  Loading  curves  will  be  developed  for  Service/commodity  groups 
on  a  monthly  basis  (and  aggregated  upward).  These  loadings 
will  detail  the  number  of  active  cases  at  each  point  in  time. 

The  number  of  active  cases  will  be  further  subdivided  into 
the  number  of  LORs,  LOOs,  LOAs,  and  open  cases  from  prior 
time  periods. 

•  With  loadings  established,  the  Air  Force  MANSAP  model  and  the 
Army  LOGCTR  model  will  be  tested  to  determine  how  closely  they 
have  predicted  the  actual  manpower  data  reported  in  the  SAMAS. 
Since  there  are  only  three  years  of  actual  SAMAS  data,  however, 
the  results  of  this  test  will  be  rather  inconclusive. 

Total  sales  and  deliveries  of  each  commodity  will  be  used  together 
with  LOR  and  case  load  details  to  develop  a  series  of  probability  density 
functions  which  describe  mixes  of  workload  in  terms  of  the  dollar  values. 

The  mixes  then  will  form  the  basis  for  a  simulation  model  which  will  fore¬ 
cast  sets  of  workloads  possible  within  the  forecast  range  of  dollar  sales 
and  deliveries.  These  forecasts,  by  subordinate  command,  will  be  aggregated 
and  adjusted  to  be  consistent  with  the  Service  totals. 

3.6  PROGRAM  PROJECTIONS 

The  results  of  our  test  and  validation  activities  can  best  be  demon¬ 
strated  by  producing  some  sample  program  projections. 

Figure  3.10  is  a  5-year  forecast  of  the  Security  Assistance  Program 
(in  current  dollars)  as  it  might  be  developed  for  use  in  the  Consolidated 
Guidance.  Appendix  G  details  the  statistics  and  plots  associated  with 
this  projection. 

Figure  3.11  is  a  5-year  forecast  of  the  Service  Security  Assistance 
Program  (in  current  dollars)  for  Western  Europe  and  NATO.  Appendix  H 
details  the  statistics  and  plots  associated  with  this  projection. 
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SECTION  4 

CONCLUSIONS  AND  RECOmENDATIONS 


4 . 1  CONCLUSIONS 

^•1*1  Program  Projections 

We  have  successfully  tested  and  validated  a  wide  variety  of  program 
projection  methods  at  various  levels  of  aggregation.  In  the  process,  we 
have  narrowed  the  choice  of  method  to  a  selected  few  which  can  be  developed 
and  used  with  considerable  confidence. 


We  conclude  that  the  development  of  statistically  based  program 
projections  should  proceed  as  follows: 

4-1. 1.1  Program  Projection  (Sales) 

•  Examine  the  actual  annual  sales  data  set  to: 

Develop  the  statistics^  of  the  data. 

-  Compile  the  cumulative  sum  of  sales  data. 

Adjust  the  data  to  constant  dollars. 

•  Use  the  cumulative  sum  of  sales  data  to  determine 
regression  constants  for  a  linear  fit  in  the  form  of: 

Cumulative  Sales  =  a  +  b(Time) 

•  Compare  the  value  of  the  regression  constant  b  with  the 
mean  value  of  the  data  set. 

An  exact  correspondence  between  the  two  values 
indicates  a  perfect  fit  of  the  data  set. 

A  large  deviation  indicates  the  presence  of  some 
kind  of  long  term  variation  (cycles  or  other 
similarities  between  subsets  of  the  total  data). 


Hlaxi 


mum, 


mean,  and  minimum  values. 


and  the  standard  deviation. 
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•  Plot  the  ctunulative  sum  of  sales  data  sets.  Compare  the  plot 
of  actual  data  to  the  plot  of  regressed  data  to  see  if  the 
difference  between  the  cumulative  straight  line  regression 
and  the  actual  cumulative  data  appears  to  be  cyclical.^ 

•  Determine  the  existence  of  cycles  within  the  data  by  establish¬ 
ing  the  percents  of  trend  or  other  relationships  of  the  cyclical 
residuals. 

•  Define  the  characteristics  of  the  cycles  by  testing  for  and 
computing  the  deviation  (amplitudes)  and  frequency  (duration). 

«  Regress  the  suspected  cyclical  data  sets  of  the  cumulative 

sales  regression  equation,  one  on  the  other.  Use  a  regression 
equation  in  the  form: 

Cumulative  Sales  X  =  a  +  b (cumulative  Sales  XI) 

0  Observe  the  degree  of  correlation  obtained  and  measure  the 
forecast  accuracy  by  the  coefficient  of  determination  and 
the  standard  error. 

•  Make  the  Sales  projection  using  the  cumulative  sales  regres¬ 
sion  equation  as  modified  to  reflect  the  cyclical  relationships 
between  data  sets.  Pages  H-46-51  of  Volume  II  provides  a  de¬ 
scription  of  the  step  by  step  procedure  used  to  obtain  the 
Army  Sales  projection  for  Western  Europe  and  NATO. 


The  cumulative  sum  of  sales  regression  is  but  one  projection  methodology 
to  be  examined.  Other  curve  fits  using  different  equations  should  be 
investigated  as  well.  To  do  so,  plots  of  yearly  data  and  the  cumulative 
data  are  required.  The  cumulative  data  plot  smooths  the  yearly  sales 
differences.  Thus,  deviations  from  the  regression  line  can  be  tested 
to  see  if  the  difference  forms  a  cycle  (systematic  deviations  from  the 
regression  slope  coefficient  "b”)  or  if  the  deviations  form  a  random 
data  set.  The  existence  of  cycles  indicates  that  a  curve  fit  might  be 
found  for  the  yearly  data  in  the  form  of  some  cyclical  curve  such  as  the 
classical  sine/cosine  terms  of  the  Fourier  transform  or  the  Hoerl's 
logarithmic-exponential  construct  properly  applied  to  the  data.  The 
steps  which  follow  determine  which  forms  of  equation  should  be  tested. 
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4. 1.1. 2  Program  Projection  (Deliveries) 

For  a  Deliveries  projection  based  on  the  Deliveries  history: 


•  Examine  the  actual  annual  deliveries  data  set  to  develop 
the  data  statistics  and  determine  the  existence  of  cycles. 

•  When  cycles  are  clearly  defined,  test  the  cyclical  data  by 
regressing  one  set  on  another  to  verify  that  relationship. 

•  Use  the  data  with  an  appropriate  curve  form  and  obtain 
regression  constants  for  the  cyclical  deliveries  data. 

If  the  data  continue  to  follow  the  logarithmic- 

exponential  form,  the  equation  is; 

log  Deliveries  =  log  a  +  b(log  Time)  +  c(Tlme). 

•  When  all  relationships  have  been  determined,  make  the 
Deliveries  projection  using  the  logarithmic-exponential 
regression  equation  (if  appropriate)  to  modify  any  other 
equation  developed  to  reflect  the  regression  relationships 
between  cyclical  data  sets. 

For  a  Deliveries  projection  based  on  a  Sales  projection: 

•  Regress  the  actual  annual  deliveries  data  set  on  the 
actual  annual  sales  data  set. 

•  Observe  the  relationships  obtained  and  establish  re¬ 
gression  constants  for  a  Deliveries /Sales  model  in 
the  form: 

D^.  =  f(Sj.)  +  f(S^_;^)  +  f(S^_2)  + 

•  When  the  regression  provides  a  sufficiently  high  coefficient 
of  determination  and  the  regression  constants  are  established, 
make  the  Deliveries  projection  using  the  developed  equation. 
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4. 1.1. 3  Program  Projections  (General) 

A  technique  remains  to  be  developed  to  integrate  each  individual 
program  projection  with  all  other  projections  at  various  levels  of  aggre¬ 
gation  to  ensure  that  all  program  projections  are  consistent. 

Each  program  projection  produced  must  be  subjected  to  expert 
examination  for  affirmation  and/or  revision  because  the  projection 
methodologies  rely  on  the  observation  of  statistical  cycles  which  appear 
to  occur  within  the  30-year  history  of  the  FMS  and  MAP  data  bases. 

As  we  gain  experience  with  both  the  projection  methods  and  the 
forecast  equations  established  by  our  work  to  date  and  as  we  gain  incre¬ 
mental  additions  of  program  history  to  the  time  series  data  base,  con¬ 
fidence  in  the  projected  Sales  and  Deliveries  programs  made  by  these 
methods  will  grow. 

4.2.3  Manpower  Forecasts 

In  our  Phase  II  report,  we  recommended  that  the  test  and  validataion 
of  a  macro  manpower  forecasting  methodology  not  be  undertaken.  However, 
comments  by  the  Services  indicated  a  desire  to  continue  and  complete  the 
test  and  validation  of  a  macro  forecasting  methodology.  We  have  devoted 
considerable  effort  to  the  test  of  a  macro  forecasting  methodology  and 
conclude  that  the  development  of  a  macro  method  of  manpower  forecast,  if 
desired,  should  proceed  as  follows: 

•  Examine  the  actual  annual  sales  data  set 

•  Examine  the  actual  annual  deliveries  data  set 

•  Examine  the  actual  annual  manpower  data  set 

Regress  the  actual  manpower  data  on  actual  sales  and  actual  deliveries 
to  determine  the  regression  constants  for  a  manpower  forecast  equation 
in  the  form: 


M^  =  a  +  b(S^)  +  c(D^) 
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If  the  regression  produces  a  satisfactory  coefficient  of  determina¬ 
tion  and  standard  error,  use  the  manpower  forecast  equation  developed  for 
the  year  in  question  by  inserting  the  projected  Sales  and  Deliveries  for 
that  year  to  produce  the  Manpower  forecast  in  man-years. 

We  conclude  also  that  the  three  years  of  actual  and  three  years  of 
estimated  manpower  data  available  now  within  the  SAMAS  reporting  system 
are  inadequate.  A  manpower  forecast  developed  using  the  gross  relationships 
between  the  dollar  volume  of  sales  and  deliveries  and  only  three  years  of 
manpower  history  will  fail. 

The  limitation  of  only  three  years  manpower  history  data  that  mili¬ 
tates  so  heavily  against  the  utilization  of  a  macro  forecasting  method  also 
affects  the  other  methods.  Yet,  in  cases  where  estimating  equations  have 
already  been  developed  to  estimate  manpower  requirements  based  on  work  units, 
there  appears  to  be  a  reasonable  opportunity  to  develop  manpower  forecasting 
methodologies.  We  conclude  that 

•  the  potential  to  provide  a  useful  -intermedi-ate  fore¬ 
casting  method  is  rather  high  given  the  capability  to 
relate  projected  dollar  sales  and  work  factor  loadings. 

•  the  potential  to  provide  a  useful  micro  forecasting 
method  is  high  but  will  also  rely  heavily  on  the  estab¬ 
lishment  of  work  factor  loadings.  We  expect  to  forecast 
manpower  using  a  micro  method  with  at  least  the  same 
accuracy  as  we  forecast  the  work  factor  loadings. 

In  either  case,  the  overall  history  to  be  used  in  the  development 
of  work  factor /manpower  requirement  relationships  is  not  limited  to 
three  years  of  SAMAS  data.  Rather,  it  will  be  based  on  the  extensive 
survey  and  analysis  work  already  done  by  the  Services  to  develop  manpower 
estimating  procedures.  If  that  work  can  be  successfully  integrated  with 
the  detailed  sales  and  deliveries  data  to  generate  work  factors,  a  firmly 
based  set  of  forecasting  methodologies  seems  possible. 
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We  are  vigorously  pursuing  the  intermediate  and  micro  forecasting 
methodologies  and  are  optimistic  that  one  or  both  will  prove  suitable 
as  manpower  forecasting  tools. 

4.1.3  Summation 

Our  test  and  validation  efforts  have  been  slowed  significantly  by 
a  series  of  data  processing  delays.  Nonetheless,  considerable  progress 
has  been  made  and,  so  far,  all  results  are  positive. 

•  Usable  procedures  for  program  projection  have  been  success¬ 
fully  established. 

•  Feasible  procedures  for  macro  manpower  forecasting  have 
been  established  (given  a  suitable  historical  data  base). 

•  Prospects  for  the  successful  development  of  procedures 
for  both  intermediate  and  micro  manpower  forecasting  are 
encouraging  (based  on  our  analysis  to  date) . 

4 . 2  RECOMMENDATIONS 

4.2.1  Program  Projections 

We  recommend  that: 

•  No  further  test  or  validation  of  program  projection  method¬ 
ologies  be  undertaken  except  as  it  is  integral  to  the  test 
of  a  manpower  forecasting  methodology. 

•  Test  and  validation  of  a  linear  or  dynamic  programing 
technique  for  integrating  all  individual  program  projections 
to  ensure  both  consistency  and  simultaneity  be  undertaken 
after  completion  of  this  phase  of  the  project. 

•  Test  and  validation  of  methodologies  for  replicating 
of  simulating  the  life  cycle  of  letters  of  request  be 
continued  and  completed. 
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4.2.2  Manpower  Forecasts 
We  reconnnend  that: 


•  No  further  test  or  validation  of  a  macro  method  for 
manpower  forecasting  be  undertaken  except  as  it  is 
integral  to  any  other  test  and  validation  activity. 

•  Test  and  validation  of  methods  of  establishing  com¬ 
modity/work  factor /manpower  relationships  be  continued 
and  completed. 

•  Test  and  validation  of  intermediate  methods  for  manpower 
forecasting  based  on  established  commodity/work  factor/ 
manpower  relationships  be  continued  and  completed. 

•  Test  and  validation  of  a  micro  method  for  manpower  fore¬ 
casting  based  on  the  LOR  life  cycle  be  continued  and 
completed. 

•  Test  and  validation  of  a  micro  method  for  manpower  fore¬ 
casting  based  on  Sales  and  Deliveries  projections  and 
work  factor  relationships  be  undertaken  after  completion 
of  this  phase  of  the  project. 

4.2.3  Other  Recommendations 

We  recommend  that: 

•  The  final  Study  Documentation  Report  for  this  phase  concen¬ 
trate  on  specific  recommendations  for  the  selection  of 
methodologies  and  the  development  of  a  program  projection 
and  manpower  forecasting  system. 

•  The  development  of  selected  and  recommended  methodologies 

be  deferred  until  completion  of  the  final  Study  Documentation 
Report  for  this  phase. 


